Modern storage systems, such as XtremIO, leverage InfiniBand networks to connect storage controllers to each, leveraging remote direct memory access (RDMA) high speed connection. InfiniBand is a specification for a type of communications link for data flow between processors and input/output (TO) devices. XtremIO data path processes are connected via RDMA and pass requests and metadata between the different processes.
A brick configuration in XtremIO system is two storage controllers connected to a shared solid state drive (SSD) drawer. In the basic configuration of a single brick, the storage controllers are connected to each other back-to-back via InfiniBand cables. Each storage controller has two ports and each cable connects each port to its adjacent port.
When expanding to a multi-brick system, two InfiniBand switches are needed in order to interconnect several storage controllers. Each port is then connected to a different switch, and the switches themselves are connected via two ports.